Skip to content

OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures#1166

Draft
krickert wants to merge 11 commits into
apache:mainfrom
ai-pipestream:light-stemmers
Draft

OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures#1166
krickert wants to merge 11 commits into
apache:mainfrom
ai-pipestream:light-stemmers

Conversation

@krickert

Copy link
Copy Markdown
Contributor

STACKED on #1163 (stemmer-factory): this branch builds on that PR's StemmerFactory contract, and since that head lives on a fork it cannot be this PR's base ref. Until #1163 merges, the diff here shows its commits too; only the last commit (592380e) is this PR. After #1163 lands, the branch rebases onto main and the diff collapses to this change alone.

Adds sixteen stemmers in a new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, and Russian, and the English minimal stemmer. These fill the tier between no stemming and the aggressive Snowball algorithms: they remove plural and inflectional endings with far less conflation. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice in each source file and in the distribution LICENSE.

Every stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input fails loudly.

Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, complete lists for Norwegian) regenerated by running the originals, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.

JIRA to follow; title gets the key once filed.

krickert added 5 commits July 8, 2026 15:05
…d SharingStemmer

SnowballStemmer now routes each call to a per-thread generated engine via
OwnerOrPerThreadState, the same pattern as the thread-safe *ME components,
so one instance can be shared across threads without touching the
generated Snowball code. StemmerFactory (api) captures stemmer
configuration as an immutable, thread-safe seam for stateful engines;
SharingStemmer adapts a factory into a shareable Stemmer for
implementations that are not thread-safe themselves (e.g. PorterStemmer).
NormalizationProfile.matchingAnalyzer() and TermAnalyzer are now safe to
share across threads when stemming.
…unit tests

MultiThreadedStemmerEval hammers shared SnowballStemmer instances for
every algorithm, a SharingStemmer-wrapped PorterStemmer, and every
NormalizationProfile matching analyzer from 8 threads, comparing each
result against a single-threaded reference, mirroring
MultiThreadedToolsEval. The unit tests now also cover all 21 algorithms
under concurrency, the owner thread stemming concurrently with pool
threads, virtual threads (a fresh thread per task), and Porter sharing.
…atch baseline

Measures the thread-safe SnowballStemmer shared across threads and one
per thread against a replica of the pre-patch plain-field
implementation, at 1/8/32 threads. Results in BENCHMARKS.md: the
OwnerOrPerThreadState overhead is within error bars up to 8 threads and
only shows (~1.5x) at full 32-thread saturation.
…to-stem mappings

Natural text is Zipf-distributed, so the same words are stemmed
constantly. CachingStemmer wraps a StemmerFactory with a bounded
per-thread LRU (default 1024 entries) following the same
OwnerOrPerThreadState pattern as the *ME components: no cross-thread
sharing, thread-safe regardless of the delegate. On the JMH zipf
workload the cache is a ~34x throughput multiplier; on a cache-hostile
uniform workload over 8x the cache capacity it still breaks even.
Results and a corrected JMH invocation are documented in BENCHMARKS.md.
…ngStemmer

Builder.stem(StemmerFactory) now wraps the factory in a CachingStemmer
(default capacity), with a stem(StemmerFactory, int) overload for
explicit cache sizing, and NormalizationProfile.matchingAnalyzer()
builds through the factory again so profile analyzers get the per-thread
stem cache for free. Analyzer results are unchanged; repeated words
resolve from the cache instead of being re-stemmed.
@krickert krickert changed the title UniNE light and minimal stemmer tiers with vocabulary parity fixtures OPENNLP-1886 - UniNE light and minimal stemmer tiers with vocabulary parity fixtures Jul 10, 2026
…-indirection factory products

SharingStemmer forwards stemAll to its per-thread delegate so a
multi-output engine keeps its full result list through the wrapper, and
all three thread-routing stemmers expose clearThreadLocalState(),
mirroring the thread-safe ME components, so pooled threads can release
their engine and cache instead of retaining them until the instance is
collected. The caching javadoc and BENCHMARKS.md now state that the
memoization is keyed to the physical thread and what that means on a
virtual-thread-per-task executor.

SnowballStemmerFactory products are now thread-confined stemmers driving
their engine through a plain field, so the one-stemmer-per-thread pattern
and the per-thread delegates inside the caching and sharing wrappers pay
none of the shared-instance thread routing. The SnowballStemmer
constructor validates algorithm and repeat like the factory, validation
is IllegalArgumentException across the new surface, the LRU map is sized
for its load factor so a full cache never rehashes, and StemmerFactory
documents that it is a plain supplier, not one of the BaseToolFactory
tool factories; its one-shot stem convenience method is gone.

Tests cover the multi-output forwarding, the repeat pass-through with a
witness word that stems differently at repeat one and two, the owner
cache reset and worker-thread delegate release behind
clearThreadLocalState, and the constructor validation.
@krickert krickert changed the title OPENNLP-1886 - UniNE light and minimal stemmer tiers with vocabulary parity fixtures OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures Jul 10, 2026
@krickert krickert self-assigned this Jul 10, 2026
krickert added 3 commits July 11, 2026 15:06
…g, cache controls

The Stemmer and StemmerFactory interfaces state their contracts without
implementation or threading claims, and null words throw
IllegalArgumentException across the API (Snowball, factory products,
caching and sharing wrappers, Porter), pinned by tests. SnowballStemmer
now composes SharingStemmer so the per-thread routing pattern lives in
one class. CachingStemmer gains clearCache() for the calling thread,
interns cached stems through StringInterners, and both stateful classes
document the container release path of the *ME components.
…erflow

The cached stems are no longer routed through the JVM-wide interner: its
default implementation holds strong references forever, so open-vocabulary
input would grow without bound and defeat the documented per-thread cache
cap. The LRU alone keeps the bound honest. The LRU table sizing is computed
in double arithmetic because the int expression overflows to a negative
capacity near Integer.MAX_VALUE; a test pins construction at the extreme.
clearCache() documents that it initializes state for a thread that has not
stemmed yet.
…ures

Sixteen stemmers in the new opennlp.tools.stemmer.light package: light and
minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk
varieties), Swedish, plus light stemmers for Finnish, Hungarian, Italian,
Portuguese, Russian and the English minimal stemmer. Adapted from Apache
Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's
BSD notice in each source file and in the distribution LICENSE. These fill
the tier between no stemming and the aggressive Snowball algorithms.

Every stemmer is stateless and thread-safe, implements Stemmer, and is its
own StemmerFactory. Input is expected lowercase, matching the algorithms'
original contract; null input fails loudly.

Parity is asserted against the original implementations: bundled fixtures
sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per
stemmer, all pairs for the small Norwegian lists) regenerated by running
the Lucene classes, so any behavioral drift fails the test. The Galician
and Portuguese minimal stemmers are excluded because they build on the RSLP
rule engine, which is its own effort.
krickert added 2 commits July 12, 2026 15:28
The review conventions keep JIRA keys out of source comments; the sizing note stands on its own.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant